Intrinsic parameters estimation in visual tracking systems

ABSTRACT

A method for adjusting camera intrinsic parameters of a multi-camera visual tracking device is described. In one aspect, a method for calibrating the multi-camera visual tracking system includes disabling a first camera of the multi-camera visual tracking system while a second camera of the multi-camera visual tracking system is enabled, detecting a first set of features in a first image generated by the first camera after detecting that the temperature of the first camera is within the threshold of the factory calibration temperature of the first camera, and accessing and correcting intrinsic parameters of the second camera based on the projection of the first set of features in the second image and a second set of features in the second image.

CLAIM OF PRIORITY

The present application is a continuation of U.S. Pat. ApplicationSerial No. 17/448,655, filed Sep. 23, 2021, which application claims thebenefit of priority to U.S. Provisional Pat. Application Serial No.63/189,935, filed May 18, 2021, both of which are hereby incorporated byreference in their entireties.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to a visualtracking system. Specifically, the present disclosure addresses systemsand methods for calibrating multiple cameras of a visual trackingsystems.

BACKGROUND

An augmented reality (AR) device enables a user to observe a scene whilesimultaneously seeing relevant virtual content that may be aligned toitems, images, objects, or environments in the field of view of thedevice. A virtual reality (VR) device provides a more immersiveexperience than an AR device. The VR device blocks out the field of viewof the user with virtual content that is displayed based on a positionand orientation of the VR device.

Both AR and VR devices rely on motion tracking systems that track a pose(e.g., orientation, position, location) of the device. A motion trackingsystem is typically factory calibrated (based on predefined/knownrelative positions between the cameras and other sensors) to accuratelydisplay the virtual content at a desired location relative to itsenvironment. However, factory calibration parameters are based onfactory conditions that are different from user operating conditions.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

To easily identify the discussion of any particular element or act, themost significant digit or digits in a reference number refer to thefigure number in which that element is first introduced.

FIG. 1 is a block diagram illustrating an environment for operating avisual tracking system in accordance with one example embodiment.

FIG. 2 is a block diagram illustrating a display device in accordancewith one example embodiment.

FIG. 3 is a block diagram illustrating a visual tracking system inaccordance with one example embodiment.

FIG. 4 is a block diagram illustrating a tracking calibration module inaccordance with one example embodiment.

FIG. 5 is a flow diagram illustrating a method for projecting featuresin accordance with one example embodiment.

FIG. 6 is a flow diagram illustrating a method for forming a temperatureprofile in accordance with one example embodiment.

FIG. 7 is a flow diagram illustrating a method for identifying intrinsicparameters in accordance with one example embodiment.

FIG. 8 illustrates an aspect of the subject matter in accordance withone embodiment.

FIG. 9 illustrates a network environment in which a head-wearable devicecan be implemented according to one example embodiment.

FIG. 10 is a block diagram showing a software architecture within whichthe present disclosure may be implemented, according to an exampleembodiment.

FIG. 11 is a diagrammatic representation of a machine in the form of acomputer system within which a set of instructions may be executed forcausing the machine to perform any one or more of the methodologiesdiscussed herein, according to one example embodiment.

DETAILED DESCRIPTION

The description that follows describes systems, methods, techniques,instruction sequences, and computing machine program products thatillustrate example embodiments of the present subject matter. In thefollowing description, for purposes of explanation, numerous specificdetails are set forth in order to provide an understanding of variousembodiments of the present subject matter. It will be evident, however,to those skilled in the art, that embodiments of the present subjectmatter may be practiced without some or other of these specific details.Examples merely typify possible variations. Unless explicitly statedotherwise, structures (e.g., structural Components, such as modules) areoptional and may be combined or subdivided, and operations (e.g., in aprocedure, algorithm, or other function) may vary in sequence or becombined or subdivided.

The term “augmented reality” (AR) is used herein to refer to aninteractive experience of a real-world environment where physicalobjects that reside in the real-world are “augmented” or enhanced bycomputer-generated digital content (also referred to as virtual contentor synthetic content). AR can also refer to a system that enables acombination of real and virtual worlds, real-time interaction, and 3Dregistration of virtual and real objects. A user of an AR systemperceives virtual content that appears to be attached or interact with areal-world physical object.

The term “virtual reality” (VR) is used herein to refer to a simulationexperience of a virtual world environment that is completely distinctfrom the real-world environment. Computer-generated digital content isdisplayed in the virtual world environment. VR also refers to a systemthat enables a user of a VR system to be completely immersed in thevirtual world environment and to interact with virtual objects presentedin the virtual world environment.

The term “AR application” is used herein to refer to a computer-operatedapplication that enables an AR experience. The term “VR application” isused herein to refer to a computer-operated application that enables aVR experience. The term “AR/VR application” refers to acomputer-operated application that enables a combination of an ARexperience or a VR experience.

The “visual tracking system” is used herein to refer to acomputer-operated application or system that enables a system to trackvisual features identified in images captured by one or more cameras ofthe visual tracking system, and build a model of a real-worldenvironment based on the tracked visual features. Non-limiting examplesof the visual tracking system include: a Visual SimultaneousLocalization and Mapping system (VSLAM), and Visual-InertialSimultaneous Localization and Mapping system (VI-SLAM). VSLAM can beused to build a target from an environment or a scene based on one ormore cameras of the visual tracking system. VI-SLAM (also referred to asa visual-inertial tracking system) determines the latest position orpose of a device based on data acquired from multiple sensors (e.g.,depth cameras, inertial sensors) of the device.

The term “intrinsic parameters” is used herein to refer to parametersthat are based on conditions internal to the camera. Non-limitingexamples of intrinsic parameters include: camera focal lengths,resolution, field of view, internal temperature of the camera, andinternal measurement offset.

The term “extrinsic parameters” is used herein to refer to parametersthat are based on conditions external to the camera. Non-limitingexamples of extrinsic parameters include: ambient temperature (e.g.,temperature of an environment in which the camera operates), andposition and orientation of the camera relative to other sensors.

AR/VR applications enable a user to access information, such as in theform of virtual content rendered in a display of an AR/VR display device(also referred to as a display device). The rendering of the virtualcontent may be based on a position of the display device relative to aphysical object or relative to a frame of reference (external to thedisplay device) so that the virtual content correctly appears in thedisplay. For AR, the virtual content appears aligned with a physicalobject as perceived by the user and a camera of the AR display device.The virtual content appears to be attached to a physical object ofinterest. In order to do this, the AR display device detects thephysical object and tracks a pose of the AR display device relative to aposition of the physical object. A pose identifies a position andorientation of the display device relative to a frame of reference orrelative to another object. For VR, the virtual object appears at alocation (in the virtual environment) based on the pose of the VRdisplay device. The virtual content is therefore refreshed based on alatest position of the device.

Cameras of the visual tracking system are subject to distortion, forexample, due to heat generated by the cameras and other componentsconnected or in proximity to the cameras. One example process is tocalibrate optical cameras to obtain the distortion model (e.g., cameraintrinsics). For AR/VR devices, calibration is a standard process to becarried on after manufacturing. This process is referred to as “factorycalibration.” Factory calibration is typically performed only oncebecause the process is time consuming. For example, during factorycalibration, the display device usually only runs the calibrationprogram in a user environment that is often different from the factorycalibration environment. In the factory calibration environment, only afew background applications operate at the display device, the displayand processors in the display device are also consuming less power (andthus generate less heat). In the real-world environment, many backgroundapplications are running, the display and processor in the displaydevice are also consuming much more power (and thus generate more heat).

The present application describes a method for identifying changes incamera distortion under various thermal conditions, and for generating atemperature-based distortion model (resulting in higher quality VSLAM).In other words, the present application describes an online cameradistortion estimation method that produces a distortion model of one ormore cameras of a display device at a given temperature condition. Inone example, a tracking calibration component turns off a first cameraof a multiple camera tracking system to reduce the temperature of thefirst camera so that the temperature of the camera is close to thetemperature of the first camera at factory calibration (also referred toas factory calibration temperature). Once the temperature of the firstcamera reaches the factory calibration temperature, the trackingcalibration component turns the first camera back on. The visualtracking system uses only the first camera in a 6DOF (degrees offreedom) tracking to gather 3D information (e.g., features) about itsenvironment. In one example, the visual-inertial tracking systemoperates as a mono VI-SLAM system using only the first camera. Featuresdetected by the first camera are projected onto an image of the secondcamera. The visual-inertial tracking system identifies detected features(on the second camera) that correspond to the projected features (fromthe first camera). The tracking calibration component generates atemperature distortion model that identifies distortions based on thecamera temperature, and the pairs of projected and detected features.The tracking calibration component can then determine intrinsicparameters of the second camera for a specific temperature based on thetemperature distortion model. The visual-inertial tracking systemadjusts and corrects the features detected by the second camera with theintrinsic parameters of the second camera that is operating at thespecific temperature.

In one example embodiment, the present application describes a methodfor adjusting camera intrinsic parameters of a multi-cameravisual-inertial tracking device. In one aspect, a method for calibratingthe multi-camera visual tracking system includes disabling a firstcamera of the multi-camera visual tracking system while a second cameraof the multi-camera visual tracking system is enabled, detecting a firstset of features in a first image generated by the first camera afterdetecting that the temperature of the first camera is within thethreshold of the factory calibration temperature of the first camera,and accessing and correcting intrinsic parameters of the second camerabased on the projection of the first set of features in the second imageand a second set of features in the second image.

As a result, one or more of the methodologies described hereinfacilitate solving the technical problem of calibrating camera intrinsicparameters based on operating conditions that are different from factoryconditions. The presently described method provides an improvement to anoperation of the functioning of a computer by providing further accuratecalibration computation to enhance a VI-SLAM pose estimation.Furthermore, one or more of the methodologies described herein mayobviate a need for certain efforts or computing resources. Examples ofsuch computing resources include Processor cycles, network traffic,memory usage, data storage capacity, power consumption, networkbandwidth, and cooling capacity.

FIG. 1 is a network diagram illustrating an environment 100 suitable foroperating an AR/VR display device 106, according to some exampleembodiments. The environment 100 includes a user 102, an AR/VR displaydevice 106, and a physical object 104. The user 102 operates the AR/VRdisplay device 106. The user 102 may be a human user (e.g., a humanbeing), a machine user (e.g., a computer configured by a softwareprogram to interact with the AR/VR display device 106), or any suitablecombination thereof (e.g., a human assisted by a machine or a machinesupervised by a human). The user 102 is associated with the AR/VRdisplay device 106.

The AR/VR display device 106 includes a computing device having adisplay such as a smartphone, a tablet computer, or a wearable computingdevice (e.g., watch or glasses). The computing device may be hand-heldor may be removable mounted to a head of the user 102. In one example,the display includes a screen that displays images captured with thecameras of the AR/VR display device 106. In another example, the displayof the device may be transparent such as in lenses of wearable computingglasses. In other examples, the display may be non-transparent,partially transparent, or partially opaque. In yet other examples, thedisplay may be wearable by the user 102 to completely or partially coverthe field of vision of the user 102.

The AR/VR display device 106 includes an AR application (not shown) thatcauses a display of virtual content based on images detected with thecameras of the AR/VR display device 106. For example, the user 102 maypoint multiple cameras of the AR/VR display device 106 to capture animage of the physical object 104. The physical object 104 is within afield of view 112 of a first camera (not shown) of the AR/VR displaydevice 106 and within a field of view 114 of a second camera (not shown)of the AR/VR display device 106. The AR application generates virtualcontent corresponding to an identified object (e.g., physical object104) in the image and presents the virtual content in a display (notshown) of the AR/VR display device 106.

The AR/VR display device 106 includes a visual tracking system 108. Thevisual tracking system 108 tracks the pose (e.g., position andorientation) of the AR/VR display device 106 relative to the real worldenvironment 110 using, for example, optical sensors (e.g., depth-enabled3D camera, image camera), inertial sensors (e.g., gyroscope,accelerometer), magnetometer, wireless sensors (Bluetooth, Wi-Fi), GPSsensor, and audio sensor. In one example, the visual tracking system 108includes a visual Simultaneous Localization and Mapping system (VSLAM)that operates with multiple cameras of the AR/VR display device 106. Inone example, the AR/VR display device 106 displays virtual content basedon the pose of the AR/VR display device 106 relative to the real worldenvironment 110 and/or the physical object 104 (as determined by thevisual tracking system 108). The visual tracking system 108 is describedin more detail below with respect to FIG. 3 .

Any of the machines, databases, or devices shown in FIG. 1 may beimplemented in a general-purpose computer modified (e.g., configured orprogrammed) by software to be a special-purpose computer to perform oneor more of the functions described herein for that machine, database, ordevice. For example, a computer system able to implement any one or moreof the methodologies described herein is discussed below with respect toFIG. 5 to FIG. 7 . As used herein, a “database” is a data storageresource and may store data structured as a text file, a table, aspreadsheet, a relational database (e.g., an object-relationaldatabase), a triple store, a hierarchical data store, or any suitablecombination thereof. Moreover, any two or more of the machines,databases, or devices illustrated in FIG. 1 may be combined into asingle machine, and the functions described herein for any singlemachine, database, or device may be subdivided among multiple machines,databases, or devices.

In one example, the AR/VR display device 106 operates withoutcommunicating with a computer network. In another example, the AR/VRdisplay device 106 communicates with the computer network. The computernetwork may be any network that enables communication between or amongmachines, databases, and devices. Accordingly, the computer network maybe a wired network, a wireless network (e.g., a mobile or cellularnetwork), or any suitable combination thereof. The computer network mayinclude one or more portions that constitute a private network, a publicnetwork (e.g., the Internet), or any suitable combination thereof.

FIG. 2 is a block diagram illustrating modules (e.g., components) of theAR/VR display device 106, according to some example embodiments. TheAR/VR display device 106 includes sensors 202, a display 204, aprocessor 208, and a storage device 206. Examples of AR/VR displaydevice 106 include a wearable computing device, a mobile computingdevice (such as a smart phone or smart tablet), a navigational device, aportable media device.

The sensors 202 include, for example, optical sensors 212 (e.g., camerasuch as a color camera, a thermal camera, a depth sensor and one ormultiple grayscale tracking cameras), an inertial sensor 214 (e.g.,gyroscope, accelerometer, magnetometer), and temperature sensors 216. Inone example, the optical sensors 212 include two or more cameras (e.g.,a first camera A 224 and a second camera B 226). The temperature sensors216 measure the temperature of the optical sensors 212, or the componentattached or connected to the optical sensors 212. The temperaturesensors 216 measure the temperature of the optical sensors 212. In oneexample, the temperature sensors 216 include a temperature sensor (notshown) disposed on a component of the AR/VR display device 106 betweenthe camera A 224 and the camera B 226. In another example, thetemperature sensors 216 include a first temperature sensor (not shown)connected to the camera A 224 and a second temperature sensor (notshown) connected to the camera B 226. In yet another example, the firsttemperature sensor is disposed on a component adjacent to the camera A224, and a second temperature sensor is disposed on a component adjacentto the camera B 226.

Other examples of sensors 202 include a proximity or location sensor(e.g., near field communication, GPS, Bluetooth, Wifi), an audio sensor(e.g., a microphone), or any suitable combination thereof. It is notedthat the sensors 202 described herein are for illustration purposes andthe sensors 202 are thus not limited to the ones described above.

The display 204 includes a screen or monitor configured to displayimages generated by the processor 208. In one example embodiment, thedisplay 204 may be transparent or semi-opaque so that the user 102 cansee through the display 204 (in AR use case). In another exampleembodiment, the display 204 covers the eyes of the user 102 and blocksout the entire field of view of the user 102 (in VR use case). Inanother example, the display 204 includes a touchscreen displayconfigured to receive a user input via a contact on the touchscreendisplay.

The processor 208 includes an AR/VR application 210 and a visualtracking system 108. The AR/VR application 210 detects the physicalobject 104 using computer vision based on the detected features of theenvironment processed by the visual tracking system 108. The AR/VRapplication 210 retrieves virtual content (e.g., 3D object model) basedon the identified physical object 104 or physical environment. The AR/VRapplication 210 renders the virtual object in the display 204. In oneexample embodiment, the AR/VR application 210 includes a local renderingengine that generates a visualization of virtual content overlaid (e.g.,superimposed upon, or otherwise displayed in tandem with) on an image ofthe physical object 104 captured by the optical sensors 212. Avisualization of the virtual content may be manipulated by adjusting aposition of the physical object 104 (e.g., its physical location,orientation, or both) relative to the optical sensors 212. Similarly,the visualization of the virtual content may be manipulated by adjustinga pose of the AR/VR display device 106 relative to the physical object104. For a VR application, the AR/VR application 210 displays thevirtual content in an immersive virtual world displayed in the display204 at a location (in the display 204) determined based on a pose of theAR/VR display device 106.

The visual tracking system 108 estimates a pose of the AR/VR displaydevice 106. For example, the visual tracking system 108 uses image dataand corresponding inertial data from the optical sensors 212 and theinertial sensor 214 to track a location and pose of the AR/VR displaydevice 106 relative to a frame of reference (e.g., detected features inthe real world environment 110). In one example embodiment, the visualtracking system 108 operates independently and asynchronously from theAR/VR application 210. For example, the visual tracking system 108operates offline without receiving any tracking request from the AR/VRapplication 210. In another example, the visual tracking system 108operates when the AR/VR application 210 is operating at the AR/VRdisplay device 106. The visual tracking system 108 identifies cameraintrinsics parameters of the optical sensors 212 and adjusts detectedfeatures in images based on the camera intrinsics parameterscorresponding to a measured temperature of the optical sensors 212.

In one example embodiment, the visual tracking system 108 turns off thecamera A 224 to reduce the temperature of the camera A 224 so that thetemperature of the camera A 224 is close to (e.g., within one degreeCelsius) the factory calibration temperature of camera A 224. Once thetemperature of the camera A 224 reaches the factory calibrationtemperature, the visual tracking system 108 turns the camera A 224 backon. The visual tracking system 108 uses only the camera A 224 in a 6DOF(degrees of freedom) tracking to gather 3D information (e.g., features)about its environment. In one example, the visual tracking system 108operates as a mono VI-SLAM system relying only on camera A 224 (and notcamera B 226). Features detected by the camera A 224 are projected ontoan image of the camera B 226. The visual tracking system 108 identifiesdetected features (from the camera B 226) that correspond to theprojected features (from the camera A 224). The visual tracking system108 forms a temperature distortion model that identifies distortionsbased on the camera temperature, and the pairs of projected and detectedfeatures. The visual tracking system 108 can then determine intrinsicparameters of the camera B 226 for a specific temperature based on thetemperature distortion model. The visual tracking system 108 can adjustand correct the features detected by the camera B 226 with the intrinsicparameters of the camera B 226 that is operating at a specifictemperature. Example components of the visual tracking system 108 isdescribed in more detail below with respect to FIG. 3 .

The storage device 206 stores virtual content 218, landmark map 220, andintrinsic parameters temperature profile 222. The virtual content 218includes, for example, a database of visual references (e.g., images ofphysical objects) and corresponding experiences (e.g., two-dimensionalor three-dimensional virtual object models). The landmark map 220 storesa map of an environment based on features detected by the visualtracking system 108. The intrinsic parameters temperature profile 222include, for example, a temperature profile of the optical sensors 212for the visual tracking system 108. In one example, the intrinsicparameters temperature profile 222 stores a temperature model thatidentifies camera intrinsic parameters for any temperature.

Any one or more of the modules described herein may be implemented usinghardware (e.g., a Processor of a machine) or a combination of hardwareand software. For example, any module described herein may configure aProcessor to perform the operations described herein for that module.Moreover, any two or more of these modules may be combined into a singlemodule, and the functions described herein for a single module may besubdivided among multiple modules. Furthermore, according to variousexample embodiments, modules described herein as being implementedwithin a single machine, database, or device may be distributed acrossmultiple machines, databases, or devices.

FIG. 3 illustrates the visual tracking system 108 in accordance with oneexample embodiment. The visual tracking system 108 includes, forexample, an odometry module 302, an optical module 304, a VSLAMapplication 306, a tracking calibration module 308, and a camera factorycalibration module 312. The odometry module 302 accesses inertial sensordata from the inertial sensor 214. The optical module 304 accessesoptical sensor data from the optical sensors 212.

The VSLAM application 306 determines a pose (e.g., location, position,orientation) of the AR/VR display device 106 relative to a frame ofreference (e.g., real world environment 110). In one example embodiment,the VSLAM application 306 includes a visual odometry system thatestimates the pose of the AR/VR display device 106 based on 3D maps offeature points from images captured with the optical sensors 212 and theinertial sensor data captured with the inertial sensor 214. The VSLAMapplication 306 is capable of operating as a stereo VI-SLAM andmonocular VI-SLAM. In other words, the VSLAM application 306 can toggleits operation between the monocular and stereo VI-SLAM without theinterruption of localization and mapping. The VSLAM application 306provides the pose information to the AR/VR application 310 so that theAR/VR application 310 can render virtual content at display locationthat is based on the pose information.

The tracking calibration module 308 identifies intrinsic parametersbased on detected features from the camera B 226 (operating a highertemperature than camera A 224) and projected features from the camera A224 (operating a lower temperature than camera A 224 or at the factorycalibration temperature). The tracking calibration module 308 forms atemperature profile model based on pairs of projected and detectedfeatures and measured camera temperature. In one example, the trackingcalibration module 308 accesses the camera intrinsic parameters from thecamera factory calibration module 312 to identify the factorycalibration temperature. The VSLAM application 306 adjusts or modifiesthe features detected by the one of the cameras based on the temperatureprofile model. The tracking calibration module 308 is described in moredetail below with respect to FIG. 4 .

FIG. 4 is a block diagram illustrating a tracking calibration module 308in accordance with one example embodiment. The tracking calibrationmodule 308 includes a camera switch controller 402, a temperature module404, a camera intrinsic parameters estimator 406, and a calibratedfeature projection module 408.

The camera switch controller 402 switches on or off the optical sensors212. In one example, the camera switch controller 402 turns off thecamera A 224 to reduce the temperature of the camera A 224. Thetemperature module 404 accesses temperature data from the temperaturesensors 216. The camera intrinsic parameters estimator 406 determinesthat the measured temperature of the camera A 224 has reached or iswithin a preset threshold (e.g., within one degree Celsius) of thefactory calibration temperature, the camera intrinsic parametersestimator 406 issues a command to the camera switch controller 402 toturn the camera A 224 back on. The VSLAM application 306 operates as amono VSLAM system relying only on camera B 226 while camera A 224 isturned off. The VSLAM application 306 operates using camera A 224 whencamera A 224 is turned back on. In one example, the visual trackingsystem 108 operates as a mono VSLAM system relying only on camera A 224(and not camera B 226).

The camera intrinsic parameters estimator 406 determines intrinsicparameters of the camera A 224 and forms a temperature profile model.The camera intrinsic parameters estimator 406 includes a projectedfeature module 410, a filter module 412, and a temperature profilemodule 414.

The projected feature module 410 detects features in an image fromcamera A 224 (after it is turned back on). Those features are projectedonto a corresponding image from camera B 226. The projected featuremodule 410 identifies detected features (from the camera B 226) thatcorrespond to the projected features (from the camera A 224).

The filter module 412 filters out pairs of detected and projectedfeatures. In one example, the filter module 412 removes outliers by (1)verifying that the changing direction between feature points is fromcenter to borders (e.g., outward) in an image, (2) verifying that thepixels range or shift is within a preset range (e.g., from the center toedge, changing from 0 pixels to ~2 pixels), (3) finding the optical flowcenter which has the maximum number of inliers, and (4) verifying thatthe pixel shifting that is closer to the optical center is less than thepixels are located further away to the center.

The temperature profile module 414 forms a temperature distortion modelthat identifies distortions based on the camera temperature, and thepairs of projected and detected features. The visual tracking system 108can then determine intrinsic parameters of the camera A 224 for aspecific temperature based on the temperature distortion model. Thevisual tracking system 108 can adjust and correct the features detectedby the camera A 224 with the intrinsic parameters of the camera A 224that is operating at a specific temperature.

The calibrated feature projection module 408 accesses the filtered pairsof projected and detected features from the filter module 412 toidentify camera intrinsics parameters based on a temperature of theoptical sensors 212 (as detected by temperature sensors 216) and thetemperature distortion model. Given the filtered projected/detectedfeature pairs, the temperature profile module 414 calculates new cameraintrinsics parameters for any temperature. For any given temperature t,the temperature profile module 414 identifies the intrinsics i, whichcan be used to project 3D features on image at location P, and the sumdistances between all the P and corresponding detected features atlocation D, where the sum is minimized. The equation shown in FIG. 15(deleted) illustrates the above-described algorithm performed at thecalibrated feature projection module 408.

The following sample pseudo-code illustrates an example of animplementation in the calibrated feature projection module 408:

\documentclass{article} \usepackage{amsmath}% limits underneath \DeclareMathOperator*{\argminA}{arg\,min}\begin{document}\begin{align} & \argminA_{i \in I, t} f(i, t) := \{ i \in I, t \mid\forall_k \in I : f(k, t) \geq f(i, t)\} \\ & f(i, t) :=\sum_{\substack{(P, D) \in S \\ j \in \{1, 2,,,n\}}} |P_{i, t}^{j}- D^{j}|\\ & i := \theta(FocalLength, PrincipalPoints, RadialDistortion)\end{align} \end{document}

FIG. 5 is a flow diagram illustrating a method 500 for projectingfeatures in accordance with one example embodiment. Operations in themethod 500 may be performed by the tracking calibration module 308,using components (e.g., modules, engines) described above with respectto FIG. 4 . Accordingly, the method 500 is described by way of examplewith reference to the tracking calibration module 308. However, it shallbe appreciated that at least some of the operations of the method 500may be deployed on various other hardware configurations or be performedby similar Components residing elsewhere.

In block 502, the camera switch controller 402 turns off camera A 224and leaves camera B 226 on. In block 504, the temperature module 404accesses real-time temperature of camera A 224 and camera B 226. Indecision block 506, the projected feature module 410 determines whetherthe temperature of camera A 224 is within a threshold range of thefactory calibration temperature of camera A 224. In block 508, theprojected feature module 410 turns on camera A 224 in response todetermining that the temperature of camera A 224 is within a thresholdrange of the factory calibration temperature of camera A 224.

In block 510, the projected feature module 410 calculates detectedfeatures in 3D space using VSLAM with camera A 224 at its factorycalibration temperature. In block 512, the projected feature module 410projects features in 3D space on camera B 226. In block 514, theprojected feature module 410 detects 2D features using camera B 226.

It is to be noted that other embodiments may use different sequencing,additional or fewer operations, and different nomenclature orterminology to accomplish similar functions. In some embodiments,various operations may be performed in parallel with other operations,either in a synchronous or asynchronous manner. The operations describedherein were chosen to illustrate some principles of operations in asimplified form.

FIG. 6 is a flow diagram illustrating a method 610 for forming atemperature profile in accordance with one example embodiment.Operations in the method 610 may be performed by the trackingcalibration module 308, using components (e.g., modules, engines)described above with respect to FIG. 4 . Accordingly, the method 610 isdescribed by way of example with reference to the tracking calibrationmodule 308. However, it shall be appreciated that at least some of theoperations of the method 610 may be deployed on various other hardwareconfigurations or be performed by similar components residing elsewhere.

In block 602, the projected feature module 410 matches pairs ofprojected feature points and detected feature points. In block 604, thefilter module 412 filters outlier feature pairs. In block 606, thetemperature profile module 414 determines intrinsics parameters atspecific temperatures for camera B 226. In block 608, the temperatureprofile module 414 forms a temperature profile of camera B 226.

FIG. 7 is a flow diagram illustrating a routine 700 for identifyingintrinsic parameters in accordance with one example embodiment.Operations in the routine 700 may be performed by the trackingcalibration module 308, using components (e.g., modules, engines)described above with respect to FIG. 4 . Accordingly, the routine 700 isdescribed by way of example with reference to the tracking calibrationmodule 308. However, it shall be appreciated that at least some of theoperations of the routine 700 may be deployed on various other hardwareconfigurations or be performed by similar components residing elsewhere.

In block 702, the temperature module 404 detects a temperature of cameraA 224. In block 704, the calibrated feature projection module 408accesses a temperature profile of camera A 224. In block 706, thecalibrated feature projection module 408 identifies intrinsic parametersbased on the measured temperature and the retrieved temperature profile.In block 708, the calibrated feature projection module 408 applies theidentified intrinsic parameters to feature projection.

FIG. 8 illustrates a graph depicting filtered pairs of projected anddetected features 802 in accordance with one embodiment.

System With Head-Wearable Apparatus

FIG. 9 illustrates a network environment 900 in which the head-wearableapparatus 902 can be implemented according to one example embodiment.FIG. 9 is a high-level functional block diagram of an examplehead-wearable apparatus 902 communicatively coupled a mobile clientdevice 938 and a server system 932 via various network 940.

head-wearable apparatus 902 includes a camera, such as at least one ofvisible light camera 912, infrared emitter 914 and infrared camera 916.The client device 938 can be capable of connecting with head-wearableapparatus 902 using both a communication 934 and a communication 936.client device 938 is connected to server system 932 and network 940. Thenetwork 940 may include any combination of wired and wirelessconnections.

The head-wearable apparatus 902 further includes two image displays ofthe image display of optical assembly 904. The two include oneassociated with the left lateral side and one associated with the rightlateral side of the head-wearable apparatus 902. The head-wearableapparatus 902 also includes image display driver 908, image processor910, low-power low power circuitry 926, and high-speed circuitry 918.The image display of optical assembly 904 are for presenting images andvideos, including an image that can include a graphical user interfaceto a user of the head-wearable apparatus 902.

The image display driver 908 commands and controls the image display ofthe image display of optical assembly 904. The image display driver 908may deliver image data directly to the image display of the imagedisplay of optical assembly 904 for presentation or may have to convertthe image data into a signal or data format suitable for delivery to theimage display device. For example, the image data may be video dataformatted according to compression formats, such as H. 264 (MPEG-4 Part10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, or the like, andstill image data may be formatted according to compression formats suchas Portable Network Group (PNG), Joint Photographic Experts Group(JPEG), Tagged Image File Format (TIFF) or exchangeable image fileformat (Exif) or the like.

As noted above, head-wearable apparatus 902 includes a frame and stems(or temples) extending from a lateral side of the frame. Thehead-wearable apparatus 902 further includes a user input device 906(e.g., touch sensor or push button) including an input surface on thehead-wearable apparatus 902. The user input device 906 (e.g., touchsensor or push button) is to receive from the user an input selection tomanipulate the graphical user interface of the presented image.

The components shown in FIG. 9 for the head-wearable apparatus 902 arelocated on one or more circuit boards, for example a PCB or flexiblePCB, in the rims or temples. Alternatively, or additionally, thedepicted components can be located in the chunks, frames, hinges, orbridge of the head-wearable apparatus 902. Left and right can includedigital camera elements such as a complementarymetal-oxide-semiconductor (CMOS) image sensor, charge coupled device, acamera lens, or any other respective visible or light capturing elementsthat may be used to capture data, including images of scenes withunknown objects.

The head-wearable apparatus 902 includes a memory 922 which storesinstructions to perform a subset or all of the functions describedherein. memory 922 can also include storage device.

As shown in FIG. 9 , high-speed circuitry 918 includes high-speedprocessor 920, memory 922, and high-speed wireless circuitry 924. In theexample, the image display driver 908 is coupled to the high-speedcircuitry 918 and operated by the high-speed processor 920 in order todrive the left and right image displays of the image display of opticalassembly 904. high-speed processor 920 may be any processor capable ofmanaging high-speed communications and operation of any generalcomputing system needed for head-wearable apparatus 902. The high-speedprocessor 920 includes processing resources needed for managinghigh-speed data transfers on communication 936 to a wireless local areanetwork (WLAN) using high-speed wireless circuitry 924. In certainexamples, the high-speed processor 920 executes an operating system suchas a LINUX operating system or other such operating system of thehead-wearable apparatus 902 and the operating system is stored in memory922 for execution. In addition to any other responsibilities, thehigh-speed processor 920 executing a software architecture for thehead-wearable apparatus 902 is used to manage data transfers withhigh-speed wireless circuitry 924. In certain examples, high-speedwireless circuitry 924 is configured to implement Institute ofElectrical and Electronic Engineers (IEEE) 902.11 communicationstandards, also referred to herein as Wi-Fi. In other examples, otherhigh-speed communications standards may be implemented by high-speedwireless circuitry 924.

The low power wireless circuitry 930 and the high-speed wirelesscircuitry 924 of the head-wearable apparatus 902 can include short rangetransceivers (Bluetooth™) and wireless wide, local, or wide area networktransceivers (e.g., cellular or WiFi). The client device 938, includingthe transceivers communicating via the communication 934 andcommunication 936, may be implemented using details of the architectureof the head-wearable apparatus 902, as can other elements of network940.

The memory 922 includes any storage device capable of storing variousdata and applications, including, among other things, camera datagenerated by the left and right, infrared camera 916, and the imageprocessor 910, as well as images generated for display by the imagedisplay driver 908 on the image displays of the image display of opticalassembly 904. While memory 922 is shown as integrated with high-speedcircuitry 918, in other examples, memory 922 may be an independentstandalone element of the head-wearable apparatus 902. In certain suchexamples, electrical routing lines may provide a connection through achip that includes the high-speed processor 920 from the image processor910 or low power processor 928 to the memory 922. In other examples, thehigh-speed processor 920 may manage addressing of memory 922 such thatthe low power processor 928 will boot the high-speed processor 920 anytime that a read or write operation involving memory 922 is needed.

As shown in FIG. 9 , the low power processor 928 or high-speed processor920 of the head-wearable apparatus 902 can be coupled to the camera(visible light camera 912; infrared emitter 914, or infrared camera916), the image display driver 908, the user input device 906 (e.g.,touch sensor or push button), and the memory 922.

The head-wearable apparatus 902 is connected with a host computer. Forexample, the head-wearable apparatus 902 is paired with the clientdevice 938 via the communication 936 or connected to the server system932 via the network 940. server system 932 may be one or more computingdevices as part of a service or network computing system, for example,that include a processor, a memory, and network communication interfaceto communicate over the network 940 with the client device 938 andhead-wearable apparatus 902.

The client device 938 includes a processor and a network communicationinterface coupled to the processor. The network communication interfaceallows for communication over the network 940, communication 934 orcommunication 936. client device 938 can further store at least portionsof the instructions for generating a binaural audio content in theclient device 938′s memory to implement the functionality describedherein.

Output components of the head-wearable apparatus 902 include visualcomponents, such as a display such as a liquid crystal display (LCD), aplasma display panel (PDP), a light emitting diode (LED) display, aprojector, or a waveguide. The image displays of the optical assemblyare driven by the image display driver 908. The output components of thehead-wearable apparatus 902 further include acoustic components (e.g.,speakers), haptic components (e.g., a vibratory motor), other signalgenerators, and so forth. The input components of the head-wearableapparatus 902, the client device 938, and server system 932, such as theuser input device 906, may include alphanumeric input components (e.g.,a keyboard, a touch screen configured to receive alphanumeric input, aphoto-optical keyboard, or other alphanumeric input components),point-based input components (e.g., a mouse, a touchpad, a trackball, ajoystick, a motion sensor, or other pointing instruments), tactile inputcomponents (e.g., a physical button, a touch screen that provideslocation and force of touches or touch gestures, or other tactile inputcomponents), audio input components (e.g., a microphone), and the like.

The head-wearable apparatus 902 may optionally include additionalperipheral device elements. Such peripheral device elements may includebiometric sensors, additional sensors, or display elements integratedwith head-wearable apparatus 902. For example, peripheral deviceelements may include any I/O components including output components,motion components, position components, or any other such elementsdescribed herein.

For example, the biometric components include components to detectexpressions (e.g., hand expressions, facial expressions, vocalexpressions, body gestures, or eye tracking), measure biosignals (e.g.,blood pressure, heart rate, body temperature, perspiration, or brainwaves), identify a person (e.g., voice identification, retinalidentification, facial identification, fingerprint identification, orelectroencephalogram based identification), and the like. The motioncomponents include acceleration sensor components (e.g., accelerometer),gravitation sensor components, rotation sensor components (e.g.,gyroscope), and so forth. The position components include locationsensor components to generate location coordinates (e.g., a GlobalPositioning System (GPS) receiver component), WiFi or Bluetooth™transceivers to generate positioning system coordinates, altitude sensorcomponents (e.g., altimeters or barometers that detect air pressure fromwhich altitude may be derived), orientation sensor components (e.g.,magnetometers), and the like. Such positioning system coordinates canalso be received over and communication 936 from the client device 938via the low power wireless circuitry 930 or high-speed wirelesscircuitry 924.

FIG. 10 is block diagram 1000 showing a software architecture withinwhich the present disclosure may be implemented, according to an exampleembodiment. The software architecture 1004 is supported by hardware suchas a machine 1002 that includes Processors 1020, memory 1026, and I/OComponents 1038. In this example, the software architecture 1004 can beconceptualized as a stack of layers, where each layer provides aparticular functionality. The software architecture 1004 includes layerssuch as an operating system 1012, libraries 1010, frameworks 1008, andapplications 1006. Operationally, the applications 1006 invoke API calls1050 through the software stack and receive messages 1052 in response tothe API calls 1050.

The operating system 1012 manages hardware resources and provides commonservices. The operating system 1012 includes, for example, a kernel1014, services 1016, and drivers 1022. The kernel 1014 acts as anabstraction layer between the hardware and the other software layers.For example, the kernel 1014 provides memory management, Processormanagement (e.g., scheduling), Component management, networking, andsecurity settings, among other functionalities. The services 1016 canprovide other common services for the other software layers. The drivers1022 are responsible for controlling or interfacing with the underlyinghardware. For instance, the drivers 1022 can include display drivers,camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flashmemory drivers, serial communication drivers (e.g., Universal Serial Bus(USB) drivers), WI-FI® drivers, audio drivers, power management drivers,and so forth.

The libraries 1010 provide a low-level common infrastructure used by theapplications 1006. The libraries 1010 can include system libraries 1018(e.g., C standard library) that provide functions such as memoryallocation functions, string manipulation functions, mathematicfunctions, and the like. In addition, the libraries 1010 can include APIlibraries 1024 such as media libraries (e.g., libraries to supportpresentation and manipulation of various media formats such as MovingPicture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC),Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC),Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group(JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries(e.g., an OpenGL framework used to render in two dimensions (2D) andthree dimensions (3D) in a graphic content on a display), databaselibraries (e.g., SQLite to provide various relational databasefunctions), web libraries (e.g., WebKit to provide web browsingfunctionality), and the like. The libraries 1010 can also include a widevariety of other libraries 1028 to provide many other APIs to theapplications 1006.

The frameworks 1008 provide a high-level common infrastructure that isused by the applications 1006. For example, the frameworks 1008 providevarious graphical user interface (GUI) functions, high-level resourcemanagement, and high-level location services. The frameworks 1008 canprovide a broad spectrum of other APIs that can be used by theapplications 1006, some of which may be specific to a particularoperating system or platform.

In an example embodiment, the applications 1006 may include a homeapplication 1036, a contacts application 1030, a browser application1032, a book reader application 1034, a location application 1042, amedia application 1044, a messaging application 1046, a game application1048, and a broad assortment of other applications such as a third-partyapplication 1040. The applications 1006 are programs that executefunctions defined in the programs. Various programming languages can beemployed to create one or more of the applications 1006, structured in avariety of manners, such as object-oriented programming languages (e.g.,Objective-C, Java, or C++) or procedural programming languages (e.g., Cor assembly language). In a specific example, the third-partyapplication 1040 (e.g., an application developed using the ANDROID™ orIOS™ software development kit (SDK) by an entity other than the vendorof the particular platform) may be mobile software running on a mobileoperating system such as IOS™, ANDROID™, WINDOWS® Phone, or anothermobile operating system. In this example, the third-party application1040 can invoke the API calls 1050 provided by the operating system 1012to facilitate functionality described herein.

FIG. 11 is a diagrammatic representation of the machine 1100 withinwhich instructions 1108 (e.g., software, a program, an application, anapplet, an app, or other executable code) for causing the machine 1100to perform any one or more of the methodologies discussed herein may beexecuted. For example, the instructions 1108 may cause the machine 1100to execute any one or more of the methods described herein. Theinstructions 1108 transform the general, non-programmed machine 1100into a particular machine 1100 programmed to carry out the described andillustrated functions in the manner described. The machine 1100 mayoperate as a standalone device or may be coupled (e.g., networked) toother machines. In a networked deployment, the machine 1100 may operatein the capacity of a server machine or a client machine in aserver-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine 1100 maycomprise, but not be limited to, a server computer, a client computer, apersonal computer (PC), a tablet computer, a laptop computer, a netbook,a set-top box (STB), a PDA, an entertainment media system, a cellulartelephone, a smart phone, a mobile device, a wearable device (e.g., asmart watch), a smart home device (e.g., a smart appliance), other smartdevices, a web appliance, a network router, a network switch, a networkbridge, or any machine capable of executing the instructions 1108,sequentially or otherwise, that specify actions to be taken by themachine 1100. Further, while only a single machine 1100 is illustrated,the term “machine” shall also be taken to include a collection ofmachines that individually or jointly execute the instructions 1108 toperform any one or more of the methodologies discussed herein.

The machine 1100 may include Processors 1102, memory 1104, and I/OComponents 1142, which may be configured to communicate with each othervia a bus 1144. In an example embodiment, the Processors 1102 (e.g., aCentral Processing Unit (CPU), a Reduced Instruction Set Computing(RISC) Processor, a Complex Instruction Set Computing (CISC) Processor,a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), anASIC, a Radio-Frequency Integrated Circuit (RFIC), another Processor, orany suitable combination thereof) may include, for example, a Processor1106 and a Processor 1110 that execute the instructions 1108. The term“Processor” is intended to include multi-core Processors that maycomprise two or more independent Processors (sometimes referred to as“cores”) that may execute instructions contemporaneously. Although FIG.11 shows multiple Processors 1102, the machine 1100 may include a singleProcessor with a single core, a single Processor with multiple cores(e.g., a multi-core Processor), multiple Processors with a single core,multiple Processors with multiples cores, or any combination thereof.

The memory 1104 includes a main memory 1112, a static memory 1114, and astorage unit 1116, both accessible to the Processors 1102 via the bus1144. The main memory 1104, the static memory 1114, and storage unit1116 store the instructions 1108 embodying any one or more of themethodologies or functions described herein. The instructions 1108 mayalso reside, completely or partially, within the main memory 1112,within the static memory 1114, within machine-readable medium 1118within the storage unit 1116, within at least one of the Processors 1102(e.g., within the Processor’s cache memory), or any suitable combinationthereof, during execution thereof by the machine 1100.

The I/O Components 1142 may include a wide variety of Components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and so on. The specific I/OComponents 1142 that are included in a particular machine will depend onthe type of machine. For example, portable machines such as mobilephones may include a touch input device or other such input mechanisms,while a headless server machine will likely not include such a touchinput device. It will be appreciated that the I/O Components 1142 mayinclude many other Components that are not shown in FIG. 11 . In variousexample embodiments, the I/O Components 1142 may include outputComponents 1128 and input Components 1130. The output Components 1128may include visual Components (e.g., a display such as a plasma displaypanel (PDP), a light emitting diode (LED) display, a liquid crystaldisplay (LCD), a projector, or a cathode ray tube (CRT)), acousticComponents (e.g., speakers), haptic Components (e.g., a vibratory motor,resistance mechanisms), other signal generators, and so forth. The inputComponents 1130 may include alphanumeric input Components (e.g., akeyboard, a touch screen configured to receive alphanumeric input, aphoto-optical keyboard, or other alphanumeric input Components),point-based input Components (e.g., a mouse, a touchpad, a trackball, ajoystick, a motion sensor, or another pointing instrument), tactileinput Components (e.g., a physical button, a touch screen that provideslocation and/or force of touches or touch gestures, or other tactileinput Components), audio input Components (e.g., a microphone), and thelike.

In further example embodiments, the I/O Components 1142 may includebiometric Components 1132, motion Components 1134, environmentalComponents 1136, or position Components 1138, among a wide array ofother Components. For example, the biometric Components 1132 includeComponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebiosignals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), identify a person (e.g., voiceidentification, retinal identification, facial identification,fingerprint identification, or electroencephalogram-basedidentification), and the like. The motion Components 1134 includeacceleration sensor Components (e.g., accelerometer), gravitation sensorComponents, rotation sensor Components (e.g., gyroscope), and so forth.The environmental Components 1136 include, for example, illuminationsensor Components (e.g., photometer), temperature sensor Components(e.g., one or more thermometers that detect ambient temperature),humidity sensor Components, pressure sensor Components (e.g.,barometer), acoustic sensor Components (e.g., one or more microphonesthat detect background noise), proximity sensor Components (e.g.,infrared sensors that detect nearby objects), gas sensors (e.g., gasdetection sensors to detection concentrations of hazardous gases forsafety or to measure pollutants in the atmosphere), or other Componentsthat may provide indications, measurements, or signals corresponding toa surrounding physical environment. The position Components 1138 includelocation sensor Components (e.g., a GPS receiver Component), altitudesensor Components (e.g., altimeters or barometers that detect airpressure from which altitude may be derived), orientation sensorComponents (e.g., magnetometers), and the like.

Communication may be implemented using a wide variety of technologies.The I/O Components 1142 further include communication Components 1140operable to couple the machine 1100 to a network 1120 or devices 1122via a coupling 1124 and a coupling 1126, respectively. For example, thecommunication Components 1140 may include a network interface Componentor another suitable device to interface with the network 1120. Infurther examples, the communication Components 1140 may include wiredcommunication Components, wireless communication Components, cellularcommunication Components, Near Field Communication (NFC) Components,Bluetooth® Components (e.g., Bluetooth® Low Energy), Wi-Fi® Components,and other communication Components to provide communication via othermodalities. The devices 1122 may be another machine or any of a widevariety of peripheral devices (e.g., a peripheral device coupled via aUSB).

Moreover, the communication Components 1140 may detect identifiers orinclude Components operable to detect identifiers. For example, thecommunication Components 1140 may include Radio Frequency Identification(RFID) tag reader Components, NFC smart tag detection Components,optical reader Components (e.g., an optical sensor to detectone-dimensional bar codes such as Universal Product Code (UPC) bar code,multi-dimensional bar codes such as Quick Response (QR) code, Azteccode, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2Dbar code, and other optical codes), or acoustic detection Components(e.g., microphones to identify tagged audio signals). In addition, avariety of information may be derived via the communication Components1140, such as location via Internet Protocol (IP) geolocation, locationvia Wi-Fi® signal triangulation, location via detecting an NFC beaconsignal that may indicate a particular location, and so forth.

The various memories (e.g., memory 1104, main memory 1112, static memory1114, and/or memory of the Processors 1102) and/or storage unit 1116 maystore one or more sets of instructions and data structures (e.g.,software) embodying or used by any one or more of the methodologies orfunctions described herein. These instructions (e.g., the instructions1108), when executed by Processors 1102, cause various operations toimplement the disclosed embodiments.

The instructions 1108 may be transmitted or received over the network1120, using a transmission medium, via a network interface device (e.g.,a network interface Component included in the communication Components1140) and using any one of a number of well-known transfer protocols(e.g., hypertext transfer protocol (HTTP)). Similarly, the instructions1108 may be transmitted or received using a transmission medium via thecoupling 1126 (e.g., a peer-to-peer coupling) to the devices 1122.

Although an embodiment has been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader scope of the present disclosure. Accordingly, the specificationand drawings are to be regarded in an illustrative rather than arestrictive sense. The accompanying drawings that form a part hereof,show by way of illustration, and not of limitation, specific embodimentsin which the subject matter may be practiced. The embodimentsillustrated are described in sufficient detail to enable those skilledin the art to practice the teachings disclosed herein. Other embodimentsmay be utilized and derived therefrom, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure. This Detailed Description, therefore, is not to betaken in a limiting sense, and the scope of various embodiments isdefined only by the appended claims, along with the full range ofequivalents to which such claims are entitled.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “invention” merelyfor convenience and without intending to voluntarily limit the scope ofthis application to any single invention or inventive concept if morethan one is in fact disclosed. Thus, although specific embodiments havebeen illustrated and described herein, it should be appreciated that anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

The Abstract of the Disclosure is provided to allow the reader toquickly ascertain the nature of the technical disclosure. It issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in a single embodiment for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter lies in less than allfeatures of a single disclosed embodiment. Thus, the following claimsare hereby incorporated into the Detailed Description, with each claimstanding on its own as a separate embodiment.

EXAMPLES

Example 1 is a method for calibrating a multi-camera visual trackingsystem comprising: disabling a first camera of the multi-camera visualtracking system while a second camera of the multi-camera visualtracking system is enabled; detecting a first set of features in a firstimage generated by the first camera after detecting that the temperatureof the first camera is within the threshold of the factory calibrationtemperature of the first camera; and access and correct intrinsicparameters of the second camera based on the first set of features.

Example 2 includes example 1, further comprising: monitoring thetemperature of the first camera; detecting that the temperature of thefirst camera is within the threshold of the factory calibrationtemperature of the first camera; turning on the first camera in responseto detecting that the temperature of the first camera is within thethreshold of the factory calibration temperature of the first camera;and accessing the first image generated by the first camera after thefirst camera is turned on.

Example 3 includes example 2, further comprising: accessing a secondimage that is generated by the second camera after the first camera isturned on; projecting the first set of features from the first image inthe second image, wherein detecting the first set of features is basedon factory calibration intrinsic parameters of the first camera.

Example 4 includes example 3, further comprising: detecting a second setof features in the second image before or after the first camera isturned on, the temperature of the second camera being higher than thefactory calibration temperature of the first camera, wherein determiningintrinsic parameters of the second camera is based on a projection ofthe first set of features in the second image and the second set offeatures.

Example 5 includes example 4, further comprising: matching pairs of theprojection of first set of features in the second image with the secondset of features, wherein determining intrinsic parameters of the secondcamera is based on the matched pairs of the projection of the first setof features in the second image and the second set of features.

Example 6 includes example 5, further comprising: filtering outliersfeature pairs from the matched pairs, wherein determining intrinsicparameters of the second camera is based on the filtered outliersfeature pairs.

Example 7 includes example 6, further comprising: identifying arelationship between the intrinsic parameters and the temperature of thesecond camera based on the filtered feature pairs; and forming atemperature profile based on the relationship.

Example 8 includes example 7, further comprising: measuring thetemperature of the first camera after the first camera is turned on, thetemperature of the first camera being higher than a factory calibrationtemperature of the first camera; identifying intrinsic parameters of thefirst camera based on the measured temperature of the first camera andthe temperature profile; and applying the identified intrinsicparameters to the projected features of the first camera.

Example 9 includes example 1, further comprising: adjusting a second setof projected features from the second camera based on the temperature ofthe second camera and the intrinsic parameters of the second camera.

Example 10 includes example 1, further comprising: storing the intrinsicparameters of the second camera in a storage device of the multi-cameravisual tracking system, wherein the multi-camera visual tracking systemincludes a multi-camera visual-inertial simultaneous localization andmapping system.

Example 11 is a computing apparatus comprising: a processor; and amemory storing instructions that, when executed by the processor,configure the apparatus to perform operations comprising: disable afirst camera of a multi-camera visual tracking system while a secondcamera of the multi-camera visual tracking system is enabled; detect afirst set of features in a first image generated by the first cameraafter detecting that the temperature of the first camera is within thethreshold of the factory calibration temperature of the first camera;and access and correct intrinsic parameters of the second camera basedon the first set of features.

Example 12 includes example 11, wherein the instructions furtherconfigure the apparatus to: monitor the temperature of the first camera;detect that the temperature of the first camera is within the thresholdof the factory calibration temperature of the first camera; turn on thefirst camera in response to detecting that the temperature of the firstcamera is within the threshold of the factory calibration temperature ofthe first camera; and access the first image generated by the firstcamera after the first camera is turned on.

Example 13 includes example 12, wherein the instructions furtherconfigure the apparatus to: access a second image that is generated bythe second camera after the first camera is turned on; project the firstset of features from the first image in the second image, whereindetecting the first set of features is based on factory calibrationintrinsic parameters of the first camera.

Example 14 includes example 13, wherein the instructions furtherconfigure the apparatus to: detect a second set of features in thesecond image before or after the first camera is turned on, thetemperature of the second camera being higher than the factorycalibration temperature of the first camera, wherein determiningintrinsic parameters of the second camera is based on a projection ofthe first set of features in the second image and the second set offeatures.

Example 15 includes example 14, wherein the instructions furtherconfigure the apparatus to: match pairs of the projection of first setof features in the second image with the second set of features, whereindetermining intrinsic parameters of the second camera is based on thematched pairs of the projection of the first set of features in thesecond image and the second set of features.

Example 16 includes example 15, wherein the instructions furtherconfigure the apparatus to: filter outliers feature pairs from thematched pairs, wherein determining intrinsic parameters of the secondcamera is based on the filtered outliers feature pairs.

Example 17 includes example 16, wherein the instructions furtherconfigure the apparatus to: identify a relationship between theintrinsic parameters and the temperature of the second camera based onthe filtered feature pairs; and form a temperature profile based on therelationship.

Example 18 includes example 17, wherein the instructions furtherconfigure the apparatus to: measure the temperature of the first cameraafter the first camera is turned on, the temperature of the first camerabeing higher than a factory calibration temperature of the first camera;identify intrinsic parameters of the first camera based on the measuredtemperature of the first camera and the temperature profile; and applythe identified intrinsic parameters to the projected features of thefirst camera.

Example 19 includes example 11, wherein the instructions furtherconfigure the apparatus to: adjust a second set of projected featuresfrom the second camera based on the temperature of the second camera andthe intrinsic parameters of the second camera.

Example 20 is a non-transitory computer-readable storage medium, thecomputer-readable storage medium including instructions that whenexecuted by a computer, cause the computer to: disable a first camera ofa multi-camera visual tracking system while a second camera of themulti-camera visual tracking system is enabled; detect a first set offeatures in a first image generated by the first camera after detectingthat the temperature of the first camera is within the threshold of thefactory calibration temperature of the first camera; and access andcorrect intrinsic parameters of the second camera based on the first setof features.

What is claimed is:
 1. A method comprising: operating a second camera ofa visual tracking system with a first camera of the visual trackingsystem disabled; detecting that a temperature of the first camera iswithin a threshold of a factory calibration temperature of the firstcamera after operating the second camera; in response to detecting thatthe temperature of the first camera is within the threshold of thefactory calibration temperature of the first camera, operating the firstcamera; detecting a first set of features in a first image generated bythe first camera after operating the first camera; and correctingintrinsic parameters of the second camera based on the first set offeatures in the first image.
 2. The method of claim 1, whereincorrecting the intrinsic parameters of the second camera furthercomprise: accessing a second image that is generated by the secondcamera after operating the first camera; and projecting the first set offeatures from the first image in the second image.
 3. The method ofclaim 1, wherein detecting that the temperature of the first camera iswithin the threshold of the factory calibration temperature of the firstcamera comprises: monitoring the temperature of the first camera;turning on the first camera in response to detecting that thetemperature of the first camera is within the threshold of the factorycalibration temperature of the first camera; and accessing the firstimage generated by the first camera after the first camera is turned on.4. The method of claim 1, wherein detecting the first set of features inthe first image is based on factory calibration intrinsic parameters ofthe first camera, wherein the method further comprises: detecting asecond set of features in a second image generated by the second cameraafter the first camera is turned on, the temperature of the secondcamera being higher than the factory calibration temperature of thefirst camera, wherein the intrinsic parameters of the second camera arebased on a projection of the first set of features in the second imageand the second set of features.
 5. The method of claim 4, furthercomprising: matching pairs of the projection of the first set offeatures in the second image with the second set of features in thesecond image, wherein the intrinsic parameters of the second camera arebased on the matched pairs of the projection of the first set offeatures in the second image and the second set of features.
 6. Themethod of claim 5, further comprising: filtering outliers feature pairsfrom the matched pairs, wherein the intrinsic parameters of the secondcamera are based on the filtered outliers feature pairs.
 7. The methodof claim 6, further comprising: identifying a relationship between theintrinsic parameters and the temperature of the second camera based onthe filtered feature pairs; and forming a temperature profile based onthe relationship.
 8. The method of claim 7, further comprising:measuring the temperature of the first camera after the first camera isturned on, the temperature of the first camera being higher than afactory calibration temperature of the first camera; identifyingintrinsic parameters of the first camera based on the measuredtemperature of the first camera and the temperature profile; andapplying the identified intrinsic parameters to the projected featuresof the first camera.
 9. The method of claim 1, further comprising:adjusting a second set of projected features from the second camerabased on the temperature of the second camera and the intrinsicparameters of the second camera.
 10. The method of claim 1, furthercomprising: storing the intrinsic parameters of the second camera in astorage device of the visual tracking system, wherein the visualtracking system includes a multi-camera visual-inertial simultaneouslocalization and mapping system.
 11. A computing apparatus comprising:one or more processors; and a memory storing instructions that, whenexecuted by the one or more processors, configure the apparatus toperform operations comprising: operating a second camera of a visualtracking system with a first camera of the visual tracking systemdisabled; detecting that a temperature of the first camera is within athreshold of a factory calibration temperature of the first camera afteroperating the second camera; in response to detecting that thetemperature of the first camera is within the threshold of the factorycalibration temperature of the first camera, operating the first camera;detecting a first set of features in a first image generated by thefirst camera after operating the first camera; and correcting intrinsicparameters of the second camera based on the first set of features inthe first image.
 12. The computing apparatus of claim 11, whereincorrecting the intrinsic parameters of the second camera furthercomprise: accessing a second image that is generated by the secondcamera after operating the first camera; and projecting the first set offeatures from the first image in the second image.
 13. The computingapparatus of claim 11, wherein detecting that the temperature of thefirst camera is within the threshold of the factory calibrationtemperature of the first camera comprises: monitoring the temperature ofthe first camera; turning on the first camera in response to detectingthat the temperature of the first camera is within the threshold of thefactory calibration temperature of the first camera; and accessing thefirst image generated by the first camera after the first camera isturned on.
 14. The computing apparatus of claim 11, wherein detectingthe first set of features in the first image is based on factorycalibration intrinsic parameters of the first camera, wherein theoperations further comprise: detecting a second set of features in asecond image generated by the second camera after the first camera isturned on, the temperature of the second camera being higher than thefactory calibration temperature of the first camera, wherein theintrinsic parameters of the second camera are based on a projection ofthe first set of features in the second image and the second set offeatures.
 15. The computing apparatus of claim 14, further comprising:matching pairs of the projection of the first set of features in thesecond image with the second set of features in the second image,wherein the intrinsic parameters of the second camera are based on thematched pairs of the projection of the first set of features in thesecond image and the second set of features.
 16. The computing apparatusof claim 15, further comprising: filtering outliers feature pairs fromthe matched pairs, wherein the intrinsic parameters of the second cameraare based on the filtered outliers feature pairs.
 17. The computingapparatus of claim 16, further comprising: identifying a relationshipbetween the intrinsic parameters and the temperature of the secondcamera based on the filtered feature pairs; and forming a temperatureprofile based on the relationship.
 18. The computing apparatus of claim17, further comprising: measuring the temperature of the first cameraafter the first camera is turned on, the temperature of the first camerabeing higher than a factory calibration temperature of the first camera;identifying intrinsic parameters of the first camera based on themeasured temperature of the first camera and the temperature profile;and applying the identified intrinsic parameters to the projectedfeatures of the first camera.
 19. The computing apparatus of claim 11,further comprising: adjusting a second set of projected features fromthe second camera based on the temperature of the second camera and theintrinsic parameters of the second camera.
 20. A non-transitorycomputer-readable storage medium, the computer-readable storage mediumincluding instructions that when executed by a computer, cause thecomputer to perform operations comprising: operating a second camera ofa visual tracking system with a first camera of the visual trackingsystem disabled; detecting that a temperature of the first camera iswithin a threshold of a factory calibration temperature of the firstcamera after operating the second camera; in response to detecting thatthe temperature of the first camera is within the threshold of thefactory calibration temperature of the first camera, operating the firstcamera; detecting a first set of features in a first image generated bythe first camera after operating the first camera; and correctingintrinsic parameters of the second camera based on the first set offeatures in the first image.